{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Signals Boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: This notebook depends upon the the Retrotech dataset. If you have any issues, please rerun the [Setting up the Retrotech Dataset](1.setting-up-the-retrotech-dataset.ipynb) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "\n",
    "from aips import display_product_search, get_engine\n",
    "from aips.spark import create_view_from_collection\n",
    "\n",
    "engine = get_engine()\n",
    "spark = SparkSession.builder.appName(\"AIPS\").getOrCreate()\n",
    "products_collection = engine.get_collection(\"products\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keyword Search with No Signals Boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 4.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load -s product_search_request aips/search_requests\n",
    "def product_search_request(query, param_overrides={}):\n",
    "    request = {\"query\": query,\n",
    "               \"query_fields\": [\"name\", \"manufacturer\", \"long_description\"],\n",
    "               \"return_fields\": [\"upc\", \"name\", \"manufacturer\",\n",
    "                                 \"short_description\", \"score\"],\n",
    "               \"limit\": 5,\n",
    "               \"order_by\": [(\"score\", \"desc\"), (\"upc\", \"asc\")]}\n",
    "    return request | param_overrides"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"demo\">\n",
       "\t<center><input style=\"width:40%\" readonly type=\"text\" name=\"q\" value=\"ipad\">\n",
       "\t<input readonly type=\"submit\" value=\"Search\">\n",
       "\t</center>\n",
       "    <div class=\"results\">\n",
       "    \t\n",
       "\n",
       "    </div>\n",
       "</div>\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/400037252074.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> iPad&#xAE; - Refurbished Digital A/V Adapter\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> iPad&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/400037252258.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> iPad&#xAE; - Refurbished USB Power Adapter\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> iPad&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/400037252616.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> iPad&#xAE; - Refurbished Dock Connector-to-VGA Adapter\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> iPad&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/400037252890.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> iPad&#xAE; - Refurbished Keyboard Dock\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> iPad&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/400037253316.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> iPad&#xAE; - Refurbished Digital Camera Connection Kit\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> iPad&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query = \"ipad\"\n",
    "request = product_search_request(query)\n",
    "response = products_collection.search(**request)\n",
    "display_product_search(query, response[\"docs\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Signals Boosts (Signals Aggregation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 4.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Aggregating Signals to Create Signals Boosts...\n",
      "Wiping \"signals_boosting\" collection\n",
      "Creating \"signals_boosting\" collection\n",
      "Status: Success\n",
      "Successfully written 197168 documents\n",
      "Signals Aggregation Completed!\n"
     ]
    }
   ],
   "source": [
    "signals_collection = engine.get_collection(\"signals\")\n",
    "print(\"Aggregating Signals to Create Signals Boosts...\")\n",
    "create_view_from_collection(signals_collection, \"signals\")\n",
    "\n",
    "signals_aggregation_query = \"\"\"\n",
    "SELECT q.target AS query, c.target AS doc, COUNT(c.target) AS boost\n",
    "FROM signals c LEFT JOIN signals q ON c.query_id = q.query_id\n",
    "WHERE c.type = 'click' AND q.type = 'query'\n",
    "GROUP BY q.target, doc\n",
    "ORDER BY boost DESC\"\"\"\n",
    "\n",
    "dataframe = spark.sql(signals_aggregation_query)\n",
    "signals_boosting_collection = \\\n",
    "    engine.create_collection(\"signals_boosting\")\n",
    "signals_boosting_collection.write(dataframe)\n",
    "print(\"Signals Aggregation Completed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search with Signals Boosts Applied"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 4.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def search_for_boosts(query, collection, query_field=\"query\"):\n",
    "    boosts_request = {\"query\": query,\n",
    "                      \"query_fields\": [query_field],\n",
    "                      \"return_fields\": [\"query\", \"doc\", \"boost\"],\n",
    "                      \"limit\": 10,\n",
    "                      \"order_by\": [(\"boost\", \"desc\")]}\n",
    "    response = collection.search(**boosts_request)\n",
    "    return response[\"docs\"]\n",
    "\n",
    "def create_boosts_query(boost_documents):\n",
    "    print(f\"Boost Documents: \\n{boost_documents}\")\n",
    "    boosts = \" \".join([f'\"{b[\"doc\"]}\"^{b[\"boost\"]}' \n",
    "                       for b in boost_documents])\n",
    "    print(f\"\\nBoost Query: \\n{boosts}\\n\")\n",
    "    display(boost_documents)\n",
    "    return boosts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Boost Documents: \n",
      "[{'query': 'ipad', 'doc': '885909457588', 'boost': 966}, {'query': 'ipad', 'doc': '885909457595', 'boost': 205}, {'query': 'ipad', 'doc': '885909471812', 'boost': 202}, {'query': 'ipad', 'doc': '886111287055', 'boost': 109}, {'query': 'ipad', 'doc': '843404073153', 'boost': 73}, {'query': 'ipad', 'doc': '635753493559', 'boost': 62}, {'query': 'ipad', 'doc': '885909457601', 'boost': 62}, {'query': 'ipad', 'doc': '885909472376', 'boost': 61}, {'query': 'ipad', 'doc': '610839379408', 'boost': 29}, {'query': 'ipad', 'doc': '884962753071', 'boost': 28}]\n",
      "\n",
      "Boost Query: \n",
      "\"885909457588\"^966 \"885909457595\"^205 \"885909471812\"^202 \"886111287055\"^109 \"843404073153\"^73 \"635753493559\"^62 \"885909457601\"^62 \"885909472376\"^61 \"610839379408\"^29 \"884962753071\"^28\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'query': 'ipad', 'doc': '885909457588', 'boost': 966},\n",
       " {'query': 'ipad', 'doc': '885909457595', 'boost': 205},\n",
       " {'query': 'ipad', 'doc': '885909471812', 'boost': 202},\n",
       " {'query': 'ipad', 'doc': '886111287055', 'boost': 109},\n",
       " {'query': 'ipad', 'doc': '843404073153', 'boost': 73},\n",
       " {'query': 'ipad', 'doc': '635753493559', 'boost': 62},\n",
       " {'query': 'ipad', 'doc': '885909457601', 'boost': 62},\n",
       " {'query': 'ipad', 'doc': '885909472376', 'boost': 61},\n",
       " {'query': 'ipad', 'doc': '610839379408', 'boost': 29},\n",
       " {'query': 'ipad', 'doc': '884962753071', 'boost': 28}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div id=\"demo\">\n",
       "\t<center><input style=\"width:40%\" readonly type=\"text\" name=\"q\" value=\"ipad\">\n",
       "\t<input readonly type=\"submit\" value=\"Search\">\n",
       "\t</center>\n",
       "    <div class=\"results\">\n",
       "    \t\n",
       "\n",
       "    </div>\n",
       "</div>\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/885909457588.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> Apple&#xAE; - iPad&#xAE; 2 with Wi-Fi - 16GB - Black\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> Apple&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/885909457595.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> Apple&#xAE; - iPad&#xAE; 2 with Wi-Fi - 32GB - Black\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> Apple&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/885909471812.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> Apple&#xAE; - iPad&#xAE; 2 with Wi-Fi - 16GB - White\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> Apple&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/843404073153.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> ZAGG - InvisibleSHIELD for Apple&#xAE; iPad&#xAE; 2 - Clear\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> ZAGG</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t\n",
       "    \t<div style=\"position: relative; width: 100%; height:auto; overflow: auto;\">\n",
       "\t    \t<div style=\"position: relative; float:left; width: 120px; margin-top:5px\">\n",
       "\t    \t\t<img style=\"width:250px; height: auto; max-height:150px\" src=\"../../data/retrotech/images/885909457601.jpg\">\n",
       "\t    \t</div>\n",
       "\t    \t<div style=\"position:relative; float:left; clear:none; width: 80%; height:auto\">\n",
       "\t    \t\t<p style=\"font-size:24px; padding-left: 50px;\"><strong>\t\tName:</strong> Apple&#xAE; - iPad&#xAE; 2 with Wi-Fi - 64GB - Black\n",
       "\t\t\t\t\t<br><strong>\t\tManufacturer:</strong> Apple&#xAE;</p>\n",
       "\t    \t\t</p>\n",
       "\t    \t</div>\n",
       "    \t</div>\n",
       "    \t\n",
       "\t\t<div style=\"position:relative; clear:both; content: ' '; display: block; height: 1px; margin-top: 10px; margin-bottom:20px\">\n",
       "\t\t\t<hr style=\"color: gray; width: 95%;\" />\n",
       "\t\t</div>\n",
       "\t\t"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query = \"ipad\"\n",
    "boost_docs = search_for_boosts(query, signals_boosting_collection)\n",
    "boosts_query = create_boosts_query(boost_docs)\n",
    "request = product_search_request(query)\n",
    "request[\"query_boosts\"] = boosts_query\n",
    "\n",
    "response = products_collection.search(**request)\n",
    "display_product_search(query, response[\"docs\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Success!\n",
    "\n",
    "You have now implemented your first AI-powered search algorithm: Signals Boosting. This is an overly simplistic implementation (we'll dive much deeper into signals boosting improvements in chapter 8), but it demonstrates the power of leveraging reflected intelligence quite well. We will dive into other Reflected Intelligence techniques in future chapters, such as Collaborative Filtering (in chapter 9 - Personalized Search) and Machine-learned Ranking (in chapter 10 - Learning to Rank).\n",
    "\n",
    "Up next: Chapter 5 - [Knowledge Graph Learning](../ch05/1.open-information-extraction.ipynb)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
